Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support sending precalculated Content-MD5 header #1867

Closed
wants to merge 2 commits into from
Closed

Support sending precalculated Content-MD5 header #1867

wants to merge 2 commits into from

Conversation

butonic
Copy link

@butonic butonic commented Aug 4, 2023 •
edited
Loading

We already have the MD5 of the content and want to send along the Content-MD5 header without minio-go having to recalculate the hash. With this PR we can set SendContentMd5: false and send our hash as UserMetadata.

@butonic butonic mentioned this pull request Aug 4, 2023
Merged
@butonic butonic changed the title Support sending precalculated Contont-MD5 header Support sending precalculated Content-MD5 header Aug 4, 2023
@butonic butonic force-pushed the support-sending-custom-content-md5 branch from 681fc80 to fe85f9d Compare August 4, 2023 12:15
harshavardhana
harshavardhana requested changes Aug 5, 2023
View reviewed changes
Copy link
Member

@harshavardhana harshavardhana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a functional_tests.go @butonic

@butonic butonic force-pushed the support-sending-custom-content-md5 branch from fe85f9d to 81a17ed Compare August 7, 2023 14:32
@butonic
Copy link
Author

butonic commented Aug 7, 2023

@harshavardhana I tried to copy and adapt an existing test. Let me know if this works for you.

@butonic butonic force-pushed the support-sending-custom-content-md5 branch from 81a17ed to 6195b80 Compare August 7, 2023 14:56
@butonic
Copy link
Author

butonic commented Aug 7, 2023

wow ... trying to run go test ./functional_tests.go gives me

# command-line-arguments
./functional_tests.go:2169:130: conversion from int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
./functional_tests.go:2169:154: conversion from int64 to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
./functional_tests.go:2844:129: conversion from int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
...

@klauspost
Copy link
Contributor

This requires multipart uploads to be disabled. I am a bit hesitant, since that isn't really obvious.

Also errors:

Error: ./functional_tests.go:11409:4: syntax error: unexpected newline in composite literal; possibly missing comma or }
Error: make: *** [Makefile:32: functional-test] Error 1
Error: Process completed with exit code 2.

@butonic
Copy link
Author

butonic commented Aug 7, 2023

hm, looking at the makefile make test uses go test ./... but does not seem ro run the functional tests. If I introduce a bug it won't be picked up by go test ./...:

➜  minio-go git:(support-sending-custom-content-md5) ✗ git diff
diff --git a/functional_tests.go b/functional_tests.go
index ed15946..0ff9653 100644
--- a/functional_tests.go
+++ b/functional_tests.go
@@ -11407,7 +11407,7 @@ func testPutObjectContentMD5() {
                "bucketName": "",
                "objectName": "",
                "size":       3,
-               "opts":       opts,
+               "opts":       opts      // introduce a bug
        }
 
        // Seed random based on current time.
➜  minio-go git:(support-sending-custom-content-md5) ✗ go test ./functional_tests.go 
# command-line-arguments
./functional_tests.go:11410:21: syntax error: unexpected newline in composite literal; possibly missing comma or }
FAIL
➜  minio-go git:(support-sending-custom-content-md5) ✗ go test ./...
ok      github.com/minio/minio-go/v7    (cached)
?       github.com/minio/minio-go/v7/pkg/encrypt        [no test files]
ok      github.com/minio/minio-go/v7/pkg/credentials    (cached)
ok      github.com/minio/minio-go/v7/pkg/lifecycle      (cached)
ok      github.com/minio/minio-go/v7/pkg/notification   (cached)
ok      github.com/minio/minio-go/v7/pkg/policy (cached)
ok      github.com/minio/minio-go/v7/pkg/replication    (cached)
ok      github.com/minio/minio-go/v7/pkg/s3utils        (cached)
ok      github.com/minio/minio-go/v7/pkg/set    (cached)
?       github.com/minio/minio-go/v7/pkg/sse    [no test files]
ok      github.com/minio/minio-go/v7/pkg/signer (cached)
ok      github.com/minio/minio-go/v7/pkg/tags   (cached)
➜  minio-go git:(support-sending-custom-content-md5) ✗ go test 
PASS
ok      github.com/minio/minio-go/v7    3.033s

that seems fishy ...

@harshavardhana
Copy link
Member

diff --git a/functional_tests.go b/functional_tests.go
index ed15946..0ff9653 100644
--- a/functional_tests.go
+++ b/functional_tests.go
@@ -11407,7 +11407,7 @@ func testPutObjectContentMD5() {
"bucketName": "",
"objectName": "",
"size": 3,

  •           "opts":       opts,

  •           "opts":       opts      // introduce a bug
  }

you are missing ,

@butonic butonic force-pushed the support-sending-custom-content-md5 branch from 6195b80 to 79a0b5f Compare August 7, 2023 15:27
@butonic
Copy link
Author

butonic commented Aug 7, 2023

yes, that was on purpose to show that a sytax error in functional_tests.go will not stop make test, which uses go run ./... from succeeding. I don't understand why ✗ go test ./functional_tests.go returns errors like

./functional_tests.go:2169:130: conversion from int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
...

while go test ./... is happy at the same time.

@butonic
Copy link
Author

butonic commented Aug 7, 2023 •
edited
Loading

I see ... the functional_tests.go has a main() and is meant to be executed ... sorry for the confusion.

@klauspost so what do we do about the multipart uploads? we could disable them ... and add an explicit ContentMD5 property to the minio.PutObjectOptions?

@butonic
Support sending precalculated Content-MD5 header
03e8414 
We already have the MD5 of the content and want to send along the Content-MD5 header without minio-go having to recalculate the hash. With this PR we can set `SendContentMd5: false` and send our hash as `UserMetadata`.

Signed-off-by: Jörn Friedrich Dreyer <[email protected]>
@butonic butonic force-pushed the support-sending-custom-content-md5 branch from 79a0b5f to 03e8414 Compare August 7, 2023 15:58
@butonic
Copy link
Author

butonic commented Aug 8, 2023

Multipart uploads are automatically used for files > 16MB (minPartSize) or the opts.PartSize. We have to calculate other hashes and can easily precalculate multiple md5hashes for every part.

@klauspost what about a ContentMD5Hashes []string in the PutObjectOptions struct. When the SendContentMd5 flag is enabled the different put implementations can skip calculating the md5 when the array index for that part is non empty (if opts.ContentMD5Hashes [partNumber-1]).

@harshavardhana
Copy link
Member

There is no way to do this in the current API style cleanly, you can use minio.Core() functionality to pass your own md5sum and also construct your own API wrappers.

That should serve the purpose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@klauspost klauspost klauspost approved these changes

@harshavardhana harshavardhana harshavardhana approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@butonic @klauspost @harshavardhana